Approximate Inference for Deep Latent Gaussian Mixtures

Deep latent Gaussian models (DLGMs) composed of density and inference networks [14]—the pipeline that defines a Variational Autoencoder [8]—have achieved notable success on tasks ranging from image modeling [3] to semi-supervised classification [6, 11]. However, the approximate posterior in these models is usually chosen to be a factorized Gaussian, thereby imposing strong constraints on the posterior form and its ability to represent the true posterior, which is often multimodal. Recent work has attempted to improve the quality of the posterior approximation by altering the Stochastic Gradient Variational Bayes (SGVB) optimization objective. Burda et al. [2] proposed an importance weighted objective, and Li and Turner [10] then generalized the importance sampling approach to a family of α-divergences. Yet, changing the optimization objective is not the only way to attenuate posterior restrictions. Instead, the posterior form itself can be made richer. For instance, Kingma et al. [7] employ full-covariance Gaussian posteriors, and Nalisnick & Smyth [13] use (truncated) GEM random variables. This paper continues this later line of work by using a Gaussian mixture latent space. We describe learning and inference for not only the traditional mixture model but also Dirichlet Process mixtures [1] (with posterior truncation). Our deep Latent Gaussian mixture model (DLGMM) generalizes previous work such as Factor Mixture Analysis [12] and Deep Gaussian Mixtures [15] to arbitrary differentiable inter-layer transformations.